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Abstract 



Discrete symmetries employed to explain flavor mixing and mass hierarchies can be associated with an enlarged 
scalar sector which might lead to exotic Higgs decay modes. In this paper, we explore such a possibility in a scenario 
with S3 flavor symmetry which requires three scalar SU{2) doublets. The spectrum is fixed by minimizing the scalar 
potential, and we observe that the symmetry of the model leads to tantalizing Higgs decay modes potentially observable 
at the CERN Large Hadron Collider (LHC). 
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1 Introduction 

43 ' 



Flavor models based on discrete symmetries are often used to address issues like large/small mixing and mild/strong mass 
hierarchies in the lepton/quark sector (for reviews, see, e.g. (T|). The permutation group S3 is an attractive such candidate 
' which was introduced in (2) and explored further in J31SJ . In this paper we study the exciting prospect that such flavor 
0^ . models can predict enlarged Higgs sectors with non-standard couplings to fermions and gauge bosons, although such a 
I/"") ' nonsupersymmetric extension does not provide any additional stability to the potential. The main motivation here arises 
from flavor issues. Its supersymmetrization would indeed bring in quantum stability, but in this work we stick to a minimal 
^sO ■ nonsupersymmetric S3 flavor scenario. 

o ; 

The motivation for choosing S3 is that it is the smallest non-abelian discrete symmetry group that contains a 
2-dimensional irreducible representation which can connect two maximally mixed generations. It is the symmetry group 
of an equilateral triangle and has three irreducible representations: 1, 1' and 2, with multiplication rules: 2x2 = l + l'+2 
and 1' x 1' = 1. Besides facilitating maximal mixing through its doublet representation, S3 provides two inequivalent 
singlet representations which play a crucial role in reproducing fermion masses and mixing. To accomplish the latter, 
three scalar SU (2) doublets are introduced, which couple to the fermions as dictated by S3 symmetry. It so happens that 
large mixing among up- and down-type quarks cancel each other in the Cabibbo matrix. Neutrino masses are separately 
generated by a type-II see-saw mechanism using scalar SU(2) triplets 0, so that the mismatch between the large mixing 
of the charged leptons and the diagonal neutrino masses translates directly into the Pontecorvo-Maki-Nakagawa-Sakata 
matrix. In this paper we do not deal with those triplets, but explore the following avenues: (i) minimization of the scalar 
potential with three scalar SU{2) doublets, two of which form an S3 doublet and the third an S3 singlet, (ii) the gauge 
and Yukawa interactions of the neutral scalars, and (Hi) different nonstandard production and decay modes of the neutral 
CP-even scalars leading to the possibility of their detection at the LHC. 

For definiteness, we study the S3 model pursued in to explain the leptonic flavor structure. We concentrate on the 
complementary aspects by exploring the scalar sector. The assignments of the fermion and scalar fields are as follows: 

(Z„,L T )e2 L e ,e c ,/i c £l r c el', 

(Q 2 ,Q 3 )e2 Qx,u c ,c c ,d c ,s c g 1 b c ,t c el', (1) 

(<Puh) g 2 03 el, 
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where the notations are standard and self-explanatory. The vacuum expectation values ( VEVs) of the three scalar doublets 
(£1,2,3 induce spontaneous electroweak symmetry breaking (SSB). 



2 Scalar potential and spectrum 



The most general S3 invariant scalar potential involving three scalar doublet fields is given by J4l[6) 

V = m 2 U\fa +<j>\ct>^+mlct>\<t>z + ) ±(<t)\fa +4<fe) 2 + ^ U\fa -4&) a + AatfiWU + y(^ 3 ) 
\s(4>ifo){4>\fa + X 6 fa\(fa<f>\ + fafa\)h + A 7 4<£i402 + A 8 4 (014^1 + 02^<£ 2 ) +h.c 



(2) 



After SSB, nine degrees of freedom are left: three neutral scalars, two neutral pseudoscalars and two charged scalars 
with two degrees of freedom each. We denote the VEVs of fa by Vi and assume the Aj's to be real. For the purpose 
of generating maximal mixing in the lepton sector, we choose the vacuum alignment v\ = v 2 = v. Once we solve the 
tadpole equations, v\ = v 2 turns out to be an extremal condition if the following relations are satisfied: 



(2Ai + X 3 )v 2 + (A 5 + A 6 + A 7 >3 + 3A 8 ot 3 , 
X A vj + 2(A 5 + A 6 + X 7 )v 2 + 2X 8 v 3 /v 3 . 



(3) 



To ensure that the chosen vacuum alignment actually corresponds to a minimum of the potential, we adjust parameters to 
make sure that the determinant of the Hessian matrix is positive. Just to recall, the Hessian is defined as the square matrix 
of second order partial derivatives of a function describing its local curvatures. In this case, the function is the scalar 
potential and the Hessian is just the mass matrix of the scalars. The positivity of the eigenvalues - see later in Eq. © 
- guarantees that the potential is minimized. We note that after SSB, the potential turns out to be a polynomial of order 
four, and its global stability in the asymptotic limit (i.e. fa — > 00) is ensured by the following set of conditions: 



Ai + A 2 > 0, 



Ai + A3 > A2, 



A 4 > 0, 



A 5 + A 6 > 0, 



A 7 >0, 



A 8 > 0. 



(4) 



We now set out to find the spectrum of the three CP-even neutral scalars. We insert the expansion fa° 



hi in 



Eq. (0 to obtain the mass matrix. After its diagonalization the weak basis scalars ft.1,2,3 are expressed in terms of the 
physical scalars /i a ,6, c as 



hi = Uib hb + Uic h c 7= h a , 

v2 

h 2 = U 2 b h b + U 2c h c + —= h a , 
v2 

^3 = U 3b h b + U 3c h c , 



(5) 



where Ua and Ui C are analytically tractable but complicated functions of AjS, v and v 3 , which we do not display. The 
condition V\ = v 2 immediately leads to U\ b = U 2 b and U\ c = U 2c . The masses of the three CP-even neutral scalars are 



m 



1 

6(c) 



4A 2 v 2 - 2A 3 w 2 - v 3 (2A7U3 + 5A 8 w) , 
1 

2^ 



[4Aiu 2 u 3 + 2\ 3 v 2 v 3 + 2A 4 w| - 2A 8 u 3 + 3A 8 w| =p 



(6) 



where 



8vv 3 ^2vvl (^2(A 5 + A 6 + A 7 ) 2 - A 4 (2Ai + A 3 )J + 2A 8 i> 4 (2A 1 + A 3 ) - 3A 4 A 8 w| 
+ 12A 8 u 2 w 2 (A 5 + A 6 + A 7 ) + 12\ 2 s v 3 v 3 l+i 2v 2 v 3 (2X 1 + A 3 ) + 2A 4 wf - 2A 8 w 3 + 3A 8 w 2 



(7) 



A few things are worth noting at this stage: 



2 



(i) Since ^2,3 are all weak SU (2) doublets, their VEVs are related as: 2v 2 + u| = w| M , where v$ M as 246 GeV. 



(ii) One of the physical scalars is given by h a = (h 2 — h\)/\2, i.e. there is no dependence on A^ gj or on the 
VEVs. This happens because S3 symmetry requires the scalar mass matrix to be of the form 




which always yields (—1, 1, 0) as one eigenvector, regardless of the values of a, b, c and d. 

(Hi) We strictly follow Eq. to ensure that the potential is bounded from below. We randomly vary the magnitude of 
the Aj's in the range [0, 1], although slightly larger (but < 47r) values of |Aj| would have still kept the couplings 
perturbative. We accept a given set of {Ai, . . . , As, v} only if it satisfies the minimization conditions. 



(iv) The difference mj; 



Am 3 /u3 is positive, and when 1)3 — > 0, i.e. 



v$m, the splitting grows enormously. 



Since the maximum value of is controlled by A, < 1, m\ becomes tachyonic when V3 — s- 0. It has been 
suggested in [4 1 that with order one Yukawa couplings, the ratio v^/v ^0.1 reproduces the correct Cabibbo angle 
in the quark sector. We require v-s/v > 0.6 to ensure that m\ stays above the accepted limit. Since hb and h c have 
similar gauge and Yukawa properties, quite different from those of h a (see discussions later), we show the mass 
splitting (m c — m&) against mb in Fig. QI a ); an d the relation between mt and m a in Fig. |TJb). 
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Figure 1: Results of a random search for allowed scalar masses for a fixed vs/v = 0.6. In the left panel (a), we exhibit the 
splitting (m c - m b ) for different choices of m b . In the right panel (b), we show the allowed range of m a . 



3 Scalar couplings to gauge and matter fields 

The kinetic terms |-D M </>i| 2 (i — 1,2,3) yield the couplings of the symmetry basis hi to and Z. Clearly, these 
couplings are modified by a factor of vi/vsm < 1 compared to their SM expressions. In terms of the mass basis scalars, 
we observe the following: (i) The coupling of hb to W + W~ (or, ZZ) is the corresponding SM coupling multiplied by 
(2vllib + vsUzb) /vsm and the corresponding factor for h c is (2vUi c + V3U3C) /vsm- (H) The scalar h a does not have 
h a ZZ or h a WW couplings, unlike the other two scalars. This can be understood as follows. The gauge couplings of hi 
arise from the linear expansion 0? = Vi + hi in the kinetic term. Since V\ = V2 = v, the combination (hi + h 2 ) will 
couple to gauge bosons as proportional to v. The orthogonal combination (h 2 — h{) that represents the physical scalar 
h a - see Eq. (0 and point ii following Eq. (0 - will not have the usual scalar-gauge-gauge vertex. The four-point h\ZZ 
and h^WW couplings will, however, exist. 
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The 53 invariant Yukawa Lagrangian, where the neutral scalars and the charged leptons/quarks are in their weak basis, 
is given by (fa's are the leptonic and g^ d s are quark Yukawa couplings) 



faee c h 3 + f 5 efi c h 3 + /> c (^ 2 + Th x ) + / 2 T c (-yu/i 2 + rht) 
+g%uu c h 3 + g%uc c h 3 + g±c c {ch 2 + thi) + g%t c (-ch 2 + thi) 
+gfdd c h 3 + gfds c h 3 + g d s c (sh 2 + bhi) + g 2 l b c (~sh 2 + bhi) + H. c. 



(8) 



The couplings of hb )C to the quarks and leptons depend on the parameters v, v 3 , A; and (or, g i ' ), while the couplings 
of h a to fermions depend only on (or, g^ d ). The physical scalar couplings to the mass basis fermions are given by 
the following Yukawa matrices, displayed for the charged leptons as an example (the structures for the quark sector are 
similar modulo Cabibbo mixing): 



The position of the zeros in the matrices deserves some attention. It turns out that h a ,b,c have off-diagonal fermion 
couplings at tree level due to the absence of any natural flavor conservation 0. The numerical entries of the Yukawa 
matrices are intimately tied to the successful reproduction of the quarks' and leptons' masses and mixings. We make three 
observations at this stage, (i) We admit that unless some S3 breaking parameters are introduced a successful reproduction 
of V c b and Vt s is problematic [8j, and also domain walls will be formed. We do not intend to cover all flavor issues. We 
simply concentrate on the scalar sector whose Lagrangian is S 3 invariant to start with, (ii) h a couples only off-diagonally 
and one of the two fermions has to be from the third generation, (iii) hb, c couple diagonally as in the SM, but also possess 
small, numerically insignificant, off-diagonal couplings involving the first two generations. 

The last two points require further clarification. In a theory with more than one SU{2) Higgs doublet, tree level flavor 
changing neutral currents (FCNC) generally exist in the scalar sector. For example, they exist in the ordinary 'two Higgs 
doublet model (2HDM)', but in the supersymmetric standard model they are avoided by the arrangement that one doublet 
couples only to the up-type fermions and the other to only the down-types. In nonsupersymmetric scenarios, in the absence 
of any natural flavor conservation, symmetry arguments have been advanced in the context of multi-Higgs models to show 
that the off-diagonal Yukawa couplings of the neutral scalars are suppressed by their relation to the off-diagonal entries of 
the Cabibbo-Kobayashi-Maskawa(CKM) matrix J5). 

In the present case, S3 symmetry, under which both scalars and fermions transform nontrivially, is instrumental in sup- 
pressing the off-diagonal couplings. To provide intuitive understanding, we take, as an example, only the two-flavor /i-r 
sector together with two neutral scalars hi and h 2 . It is not difficult to see that the combination (h 2 — hi), which corre- 
sponds to h a , couples only off-diagonally, as mentioned earlier. But the other combination (h 2 + hi), which corresponds 
to hb, c following Eq. ([5]), couples only diagonally to physical /x or r. When we consider the quark sector, /1 and r would 
be replaced by second and third generation quarks which will have CKM mixing. This will yield off-diagonal entries for 
hb.c couplings to quarks suppressed by the off-diagonal CKM elements. The same happens for off-diagonal couplings 
involving the first two generations as well. The tiny size of tree level FCNC rates in an S3 flavor model has been noticed 
also earlier, where predictions for Br (r — > 3/j,), Br (Kl — > 2e) and Br (B s — > 2/i) have been given ifTOl . In some 
setups where the fermion transformations under S3 are not appropriately adjusted, the off-diagonal Yukawa couplings 
may become order one which induce sizable neutral scalar mediated rare processes, like Kl —> A* e or Kl 2tt, at tree 
level. This requires those neutral scalars to lie beyond several TeV l6l [TTI . But in our case, once we adjust the fi/gf' d 's 
to reproduce the fermion masses and mixing, the off-diagonal Yukawa couplings are determined too. The largest of them 
corresponds to CLtRh a , which is about 0.8. The second largest off-diagonal coupling is that for s^bnha, and is about 
0.02. The next in line is p,LTRh a , whose coefficient is about 0.008. The others are orders of magnitude smaller, and 
are of no numerical significance. Although FCNC processes like Bd-Bd and B s -B s mixings proceed at tree level, the 
contributions are adequately suppressed even for light scalar mediators. 





(9) 
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4 Collider signatures 



The perturbativity condition |Aj| < 1 and the requirement m b / c > 114GeV (for which we set v^/v ~ 0.6) yields 
in the neighbourhood of 120 GeV and m c within 400 GeV - see the scatter plots in Fig.Q] Both hb and h c would decay 
into the usual ZZ, WW, bb, 77, • • • modes, but the dominant decay mode of hb (or h c ) for the case of m a < 
(or m a < rn c /2) would be into h a h a . Recall that the existing limits on the Higgs mass depend crucially on the gauge 
coupling of the Higgs. Since h a ZZ or h a WW couplings are nonexistent, the mass of h a is unconstrained, i.e. m a can be 
lower than 1 14 GeV or larger than 200 GeV. We numerically calculate the strength of the hbh a h a coupling from the set of 
acceptable parameters characterizing the potential, and introduce a parameter k which is the ratio of the hbh a h a coupling 
and the h b WW coupling. The magnitude of k depends on the choice of and V3. Assuming m a = 50 GeV, we obtain k 
in the range of (5 — 30). Just to compare with a 2HDM lfl2l for illustration, the corresponding k value, when the heavier 
Higgs weighing around 400 GeV decays into two lighter Higgs weighing 1 14 GeV each, is about 10. 

In Fig. |2a) we have plotted the branching ratio of hb — > h a h a as a function of m& for two representative values 
m a = 50,75 GeV, and for k ~ 5 and 30, which correspond to the smallest and largest k obtained from the set of accepted 
scalar parameters. We observe that till the WW or ZZ decay modes open up, the branching ratio hb — > h a h a is almost 
100%. To calculate the decay widths into the usual modes (other than h a h a ), we have used HDECAY |[T3l by appropriately 
modifying the gauge and Yukawa couplings. 

As Fig.[2|b) suggests, as long as m a < m t , h a will dominantly decay into jets, and one of them can be identified as the 
6-jet. The branching ratio of h a — > /if is, nevertheless, not negligible (about 0.1). As shown in Fig.[2|c), for m a <C m t , 
the branching ratio of t — > h a c is quite sizable, which falls with increasing m a . It may be possible to reconstruct h a from 
h a — > /if. In fact, a light h a would be copiously produced from the top decay at the LHC. On the other hand, if m a > m t , 
as can be seen again from Fig.|2b), h a decays to tc with an almost 100% branching ratio. 

If k is large, then there is an interesting twist to the failed Higgs search at LEP-2. In this case, hb —> h a h a would 
overwhelm hb —> bb, and hence the conventional search for the SM-like scalar (hb, as the lighter between hb and h c ) 
would fail. This is similar to what happens in the next-to-minimal supersymmetric models, when the lightest scalar would 
dominantly decay into two pseudoscalars, and each pseudoscalar would then decay into 2b or 2r final states. In view of 
these possible 46 or 4r Higgs signals, LEP data have been reanalyzed putting constraints on the Higgs production cross 
section times the decay branching ratios 11411151 . The possibility of the Higgs cascade decays into Aj (j = quark/gluon), 
2j + 2 photons and 4 photons has been studied too II16II17I . From a study of 46 final states, a limit m/j > 110 GeV (for 
a SM-like Higgs) has been obtained |[T6l . From all other cascade decays the limit on nth will be considerably weaker. 
Our h a has the special feature that it has only off-diagonal Yukawa couplings involving one third-family fermion. If hb 
is lighter than the top quark, it would decay as hb — > h a h a — > 2b + 2j, and into b + lj + \i + t, the latter constituting 
a spectacular signal with two different lepton flavors /t and r. The standard 2b and cascade 46 decay searches are not 
sensitive to our final states, and so a value of mb much lighter than 110 GeV is not ruled out. 



5 Conclusions 

The discrete flavor symmetry S3, besides successfully reproducing fermion masses and mixing, provides an extended 
Higgs sector having unconventional decay properties. We assume all the couplings to be real, and do not deal with the 
possibility of CP violation in this paper. The potential has been minimized requiring maximal mixing for the atmospheric 
neutrinos. In our setup, there are two scalars which are SM Higgs like, except that each of them can have a dominant decay 
into the third (hb, c — > h a h a ). The latter, i.e. h a , has no h a VV-type gauge interactions, and has only flavor off-diagonal 
Yukawa couplings with one fermion from the third generation. It is not unlikely that by evading the conventional search 
strategies, both hb and h a are already buried in the existing LEP and Tevatron data. In this analysis we have not dealt with 
the two pseudoscalars, which we leave for a future study. We urge our experimental colleagues to look for our suggested 
signals at the LHC, and perhaps also reanalyze the existing collider data. 
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Figure 2: (a) Branching ratio of h b -> h a h a for two representative values of m a , and in each case for smallest / largest 
values of k in the set of accepted scalar parameters which compares the strength of the h b h a h a coupling to the strength of 
h b WW coupling; (b) branching ratios for the decay of h a , and (c) branching ratio of the top quark decay into h a and charm 
quark. 
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